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A  theory  of  performance  measurement  for  operator  controlled 
systems  is  presented.  The  theory  permits  synthesis  of  a  system  per¬ 
formance  measure  which  scores  performance  on  successive  data  samples 
based  on  the  impact  of  the  sampled  performance  on  the  overall  summary 
of  performance.  —Since  performance  is  meaisueed  and  -  evaluated  on~each_ 
jure  dynamics  of  the  controlled  element,  i.e.,  the  aircraft,  are 
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effectively  removed  from  the  measurement  eventhough  the  pilot  (operator) 
continues  to  control  the  aircraft,  While  the  theory  directly  applies  to 
problems  where  the  performanc^Aim  iting  factors  are  known,  the  method 
has  been  extended  to  apply  to  problems  where  the  performance  limiting 
factors  are  not  known  explicitly,  but  are  known  to  be  implicit  in  the  per¬ 
formance  data,. 


t^This  report  documents  the  development  of  measures  for  aircraft 
carrier  landings  for  the  glide  path  and  angle  of  attack  control  channels. 
Flight  data  obtained  from  the  Visual  Technology  Research  Center,  Naval 
Training  Equipment  Center,  Orlando,  Florida,  was  analyzed  using  the 
measures.  The  data  on  carrier  landings  were  available  on  9-track  pnag- 
netic  tap  consisting  of  flights  by  four  subjects  each  performing  on  /2 
flights.  Each  flight  was  performed  on  a  particular  combination  of  glide 
path  error  display  and  day /night  combination.  Two  types  of  gfide  path 
displays  were  used,  resulting  in  four  treatments  (two  displays  and  two  light 
conditions).  One  display  was  the  conventional  glide  path  display  and  the  other 
was  a  command  display  which  incorporates  error  rate  information  with  glide 
path  error  presentation.  Each  subject  controlled  the  aircraft  to  a  carrier  land¬ 
ing  three  times  with  each  treatment. 

The  resulting  performance  scores  were  aggregated  by  subject,  treat¬ 
ment,  range  to  carrier  deck,  and  error  and  error  rate  cells.  The  results 
suggest  that  the  command  display  offers  improved  glide  path  control  especially 
during  day  light  conditions.  Also,  observed  differences  in  performance  at 
different  error  and  error  rate  magnitudes  suggest  that  significant  non-linear 
control  technique  may  be  exhibited  by  the  pilots. 

^In  addition  to  the  aircraft  carrier  landing  problems,  an  application  of 
system  performance  measures  to  air-to-ground  weapons  launch  problems  were 
analyzed.  The  weapon  launch  problem  is  characterized  by  the  existence  of  a 
release  hyper-surface  from  which  a  high  probability  of  kill  can  be  expected. 
Thus,  the  problem  is  not  characterized  as  having  necessarily  a  single  path 
which  must  be  flown  to  a  weapons  release  point,  but  rather,  a  confluence  of 
paths  exist  which  proceed  to  the  release  hyper-surface .  A  method  for  develop¬ 
ing  a  summary  measure  for  the  weapons  release  problem  is  presented  along 
with  an  outline  of  the  method  of  synthesizing  the  associated  system  perfor-  i 
mance  measure^ 

^Finally,  a  comparison  of  the  nature  and  applications  of  linear  pilot 
models  and  system  performance  measures  is  developed  ..  It  is  shown  that  the 
two  entities  are  quite  different,  one  being  a  pilot  model  the  other  being  a 
performance  measure.  However,  it  is  shown  that  the  techniques  used  for  the 
system  performance  measure  can  be  adapted  to  the  development  of  a  non-linear 
pilot  model,  and  when  that  is  done,  the  data  available  for  linear  pilot  models 
can  be  used  to  establish  that  portion  of  the  non-linear  model  which  reflects 
linear  performance.  Development  of  the  non-linear  model  is  extehded  from  a 
linear  control  range  to  represent  non-linear  control  policies.  This  method 
permits  use  of  the  wealth  of  information  collected  over  the  years  concerning 
linear  operator  response  and  it  permits  incorporation  of  non-linear  response 
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This  is  the  final  report  on  Contract  N 6 1 339-80-C-0 1 32 
under  subcontract  for  Task  1  to  Vreuls  Research  Corporation. 

The  period  of  performance  of  the  contract  was  from  23  September 
1980  to  22  January  1982. 

The  objective  is  to  evaluate  an  approach  to  performance 
measurement  that  removes  aircraft  dynamics  from  the  pilot/system 
performance  data.  A  further  objective  is  to  make  recommendations 
as  to  the  usefulness  of  this  technique  as  a  means  for  assessing  the 
effects  of  simulator  visual  system  variables  on  pilot  performance  for: 

a.  Carrier  landings. 

b.  Air-to-ground  weapons  delivery. 

The  work  task  requirements  were  as  follows: 

a.  Provide  assistance  to  Visual  Technology  Research 
Simulator  personnel  for  the  preprocessing  of  existing 
carrier  landing  data  which  was  provided  by  the  Govern¬ 
ment  in  a  form  compatible  with  PMA's  LSI  11  computer. 

b.  Analyze  the  carrier  landing  data  with  respect  to  vertical 
and  longitudinal  axis  performance  (glide  slope,  angle  of 
attack,  and  associated  pilot's  control  inputs). 

c.  Compare  and  contrast  results  obtained  from  this  analysis 
with  the  results  obtained  from  separate  and  independent 
analyses  of  the  same  data  set.  The  purpose  is  to 
determine  whether  the  technique  yields  information 
leading  to  the  same  or  different  conclusions  with  regard 
to  the  relative  and  absolute  effects  of  each  independent 
variable  manipulated  in  the  experiment. 
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Review  of  the  results  of  the  air  to  ground  performance 
measurement  analysis  task  (Task  2)  and  make  recommenda¬ 
tions  of  the  applicability  of  the  method  to  that  task 
environment. 

Prepare  technical  reports  summarizing  the  effort  under 
this  task  in  providing  conclusions  and  recommendations. 


Background:  A  Theory  of  Performance  Measures* 

A  theoretical  model  based  on  control  theory  has  been 
developed  to  identify  the  factors  that  should  be  determined  in  both 
theoretical  and  empirically-based  performance  measures.  Use  of  the 
theory  leads  to  development  of  comprehensive  and  sensitive  measures. 

The  theory  of  performance  measurement  introducted  by 
Connelly  &  Schuler  (1 969)  is  used  here  to  develop  a  measurement 
of  the  overall  task  performance  in  terms  of  the  individual  subtask 
performance  effects.  This  theory  was  first  applied  to  flight  control 
problems  in  which  the  factors  limiting  performance  originated  in 
the  hardware  and  were  known.  It  was  recently  extended  (Connelly, 
Comeau,  &  Steinhetser,  1981)  to  permit  its  application  to  team- 
computer  systems  where  the  factors  limiting  performance  are 
not  always  known  explicitly,  but  are  known  to  exist. 

Since,  in  many  human  performance  problems,  the  factors 
limiting  performance  are  not  always  explicitly  known,  demonstrations 
of  task  performance  at  various  performance  levels  that  exhibit 
the  effects  of  those  limiting  factors  must  be  used  to  develop  the 
performance  measures .  This  empirically-based  method  for  developing 
measures  is  described  by  Connelly,  Bourne,  Loental,  &  Knoop  (1974) 
and  is  the  foundation  of  the  MAP  computer  processor.  MAP  extracts 
information  from  the  performance  demonstration  data  and  then 
constructs  the  performance  measure. 

Before  presenting  the  performance  measurement  theory, 
it  is  necessary  to  first  define  two  types  of  tasks  ana  two  types  of 
performance  measures. 

Classification  of  Tasks. 

The  goal-oriented  or  "terminal" task  begins  with  a  variety  of 
initial  conditions  and  ends  when  a  specified  objective  is  obtained.  An 
entire  task  might  consist  of  multiple  sequential  subtasks  for  which 
the  terminal  condition  of  one  subtask  is  the  initial  condition  for  a 
subsequent  subtask.  The  point  is,  that  with  terminal  tasks,  there 

*This  section  taken  from  Connelly  (1981)  (Copyrighted  by  Connelly,  1981) 
and  used  by  permission  of  the  author. 


is  always  a  specific  goal  to  be  achieved.  And  when  that  goal  has 
been  achieved,  the  task  ends. 

Continuing  tasks,  on  the  other  hand,  have  no  end  ob¬ 
jective,  but  instead  require  performance  specified  by  certain 
criteria  at  each  instant  of  time.  For  example,  the  well-known 
pursuit  tracking  tasks  used  in  psychological  studies  are 
continuous  inasmuch  as  the  participant  must  constantly  manipulate 
a  control  device  to  track  a  moving  reference  point  in  an  attempt 
to  keep  his  error  as  small  as  possible  over  the  total  test 
time.  Typically,  the  error  is  the  distance  between  a 
moving  reference  symbol  (such  as  an  "x")  and  a  tracking 
symbol  (such  as  an  "o"),  controlled  by  the  participant. 

The  test  is  conducted  for  as  long  as  the  experimenter  has 
planned,  and  a  performance  score  is  developed  as  the 
average  error  over  the  test. 

Many  applied  human  factor  problems  can  be  cast  as 
terminal  control  problems,  even  some  that  are  spoken  of  as 
continuous  tracking  tasks.  For  instance,  in  the  sighting  of 
antiaircraft  guns  the  term  "tracking  the  target"  is  often  used* 

But  this  problem  can  also  be  broken  down  into  a  sequence 
of  terminal,  goal-oriented,  tasks.  First,  the  operator  attempts 
to  acquire  the  target  in  his  sight,  a  task  which  requires 
the  reduction  of  large  errors  by  the  operator.  Then, 
once  the  target  has  been  acquired,  the  operator  attempts 
to  tracr.  it  smoothly  and  with  sufficient  lead  to  permit  a 
hit.  If  automatic  lead  prediction  circuits  are  available,  the 
operator  must  still  continue  to  track  the  target  smoothly 
until  the  initial  transients  in  the  prediction  circuits  can  die  out 
and  the  tracking  aids  can  calculate  accurate  prediction.  In 
either  case,  the  operator  must  next  commence  firing  and,  if 
tracers  are  used,  must  adjust  his  tracking  to  make  use  of 
the  tracer  information.  Finally,  when  a  hit  is  scored,  or 
the  enemy  aircraft  moves  out  of  range,  the  task  ends. 

Still  other  tasks  may  be  viewed  as  either  continuing  or 
as  terminal.  Thus,  for  example,  maintaining  aircraft  altitude  and 
heading  over  a  long  period  of  time,  such  as  in  the  constant- 
altitude  cruising  phase  of  a  lengthy  flight,  could  be  viewed  as 
either  a  continuing  tracking  task  or  as  a  terminal  problem, 
depending  on  the  availability  of  a  relief  pilot  or  of  an  autopilot, 
among  several  possibilities.  It  is  only  when  the  fundamental 
purpose  of  a  task  is  the  achievement  of  well  defined  final  conditions 
that  the  task  (or  mission)  must  be  considered  terminal. 
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Types  of  Performance  Measures 

Summary  Performance  Measures. 

A  summary  performance  measure  (SUMPM)  is  a  set  of 
rules  for  scoring  each  task  exercise.  (Note  that  in  order  to 
describe  the  measure,  it  is  necessary  to  use  two  terms:  "task" 
and  "exercise.''  A  task  is  the  set  of  subtasks  that  must  be 
completed  to  accomplish  a  goal.  An  exercise  is  one  demonstra¬ 
tion  of  the  task.  )  A  SUMPM  provides  measurement  only  of  the 
total  task  performance,  and,  as  a  result,  the  complete  information 
required  for  a  SUMPM  is  not  available  until  the  exercise  has 
been  completed.  This  property  is  a  fundamental  limitation  of  all 
SUMPM's. 

Typically,  SUMPM's  are  first  formulated  subjectively, 
and  reflect  the  judgment  of  ah  individual  or  group  concerning 
the  objective  of  the  task  and  the  factors  believed  to  be 
important  in  scoring  exercises.  These  factors  may  involve, 
for  example,  statements  about  certain  desired  terminal  ancf 
safety  conditions  that  must  be  satisfied  by  the  exercise. 

But  whatever  the  factors  are,  the  subjective  form  of  the 
SUMPM  must  then  be  converted  into  a  quantitative  form  in 
which  specific  rules  determine  the  SUMPM  value  from  the 
exercise  data. 

in  many  studies,  performance  measurement  development 
is  terminated  at  the  summary  level,  even  though  SUMPM's 
cannot  provide  sensitive  performance  discriminations,  nor  reveal 
the  effect  of  individual  and  team  technique  on  task  performance. 

System  Performance  Measure. 

The  theoretical  development  of  a  system  performance 
measure  (SYSPM)  which  reveals  the  effect  of  the  performance 
of  each  constituent  subtask  on  summary  performance,  and  which, 
as  a  result,  does  provide  sensitive  performance  discriminations, 
is  outlined  in  the  following  section  together  with  the  relationship 
between  the  SYSPM  and  the  SUMPM.  This  theory,  which  was 
developed  first  by  Connelly,  et  al.  (1969)  and  later  extended  by 
Connelly,  Zeskind,  &  Chut  j  (1 977),  recognizes  that  performance 
is  limited  both  by  machine  factors  and  by  human  factors. 
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Recognition  that  such  limiting  factors  exist,  whether  or  not  they 
are  explicitly  known,  leads  to  a  measurement  equation  that  per¬ 
mits  evaluation  of  the  effect  of  either  instantaneous  or  of  interval 
performance  on  the  performance  of  the  entire  mission. 

The  theory  has  been  successfully  applied  to  aircraft  and 
ship  control  problems  (Connelly,  1977),  and  to  team-computer 
problems  (Connelly,  et  al.,  1981). 

Once  having  selected  a  particular  SUMPM  -  that  set 
of  rules  used  for  scoring  each  (necessarily  completed)  task  exercise 
the  SYSPM  relates  in  mathematical  terms  the  effect  of  the  per¬ 
formance  of  each  constituent  subtask  on  the  SUMPM  chosen. 

With  the  SYSPM,  the  effect  on  task  summary  performance  of  the 
way  each  constituent  task  is  performed  can  thus  be  assessed. 

This  is  an  important  property  since  the  effect  of  operator  task 
performance  cannot  be  expected  to  be  uniform  over  all  team- 
computer  system  states.  The  SYSPM  function  also  has  the 
further  ability  of  being  able  to  discriminate  among  the  many 
ways  both  good  and  bad  operator(s)  performance  can  be  achieved. 
And,  since,  when  a  team  of  operators  control  the  equipment,  the 
members  can  and  do  cooperate  in  various  ways  to  achieve  high 
performance,  this  property  becomes  important  when  measuring 
the  performance  of  teams  that  are  to  be  compared. 

To  obtain  these  properties,  the  SYSPM  function  utilizes 
" reference-task  performance”  and,  in  addition,  the  effect  on  the 
summary  performance  of  deviations  from  reference-task  per¬ 
formance.  A  reference-task  performance  is  defined  here  as  the 
expected  way  each  task  will  be  performed  by  the  operator(s).  It 
includes,  for  example,  the  time  required  to  complete  the  task, 
the  number  of  errors  expected  in  attempting  the  task,  and  so  on. 

In  order  to  develop  the  expected  performance,  it  is  often  necessary 
to  class ify  an  operators)  in  terms  of  training,  experience,  and 
previous  performance. 

SYSPM's  provide  a  sensitive  and  comprehensive  per¬ 
formance  measure  for  subtasks  and  sets  of  subtasks.  By  utilizing 
reference-task  performance  and  the  significance  of  deviations  from 
such  performance,  SYSPM's  provide  information  that  enables 
them  to  identify  critical  task  components.  Critical  task 
states  in  which  accurate  or  rapid  task  performance  is  essential 
can  be  revealed  by  sun  analysis  of  the  mathematical  structure  of 
the  SYSPM  function.  Finally,  SYSPM's  permit  rapid  assessment 
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of  performance  and  provide  a  basis  for  KOR  (knowledge  of 
results)  feedback  for  training  enhancement. 

Mathematical  Development,  of  SYS  PM. 

The  term  "plant"  in  control  theory  is  used,  for  the  sake 
of  brevity,  to  refer  to  any  "object-to-be-controlled."  To  guide 
a  plant  (such  as  an  aircraft)  along  a  particular  trajectory  (to 
ccromplish  its  mission)  certain  resources  (such  as  time,  fijel, 
etc.)  must  be  used.  We  shall  see  now  how  we  measure  the 
performance  of  a  given  plant  by  attempting  to  compute  its 
optimal  (or  reference)  trajectory. 

Let  the  N  state  variables  which  define  the  Diant  be 
represented  by  the  vector  X. 

Let  the  n  control  variables  be  represented  by  the 
vector  U. 

For  each  X.  we  then  have  for  the  time  derivative 

X.  =  f{  (X,U)  for  i  *  1,  ...,  N  (1) 

where:  f.  is  (typically)  some  non-linear  differential  function 
of  the  staVe  and  control  vectors. 

As  a  rule,  the  n  control  variables  are  each  limited  in 
some  way.  For  instance,  a  ship’s  rudder  angle  is  limited  to 
some  maximum  value,  as  is  the  rate  of  change  of  the  rudder's 
deflection.  Likewise,  the  thrust  of  an  aircraft  engine  has  a 
maximum  value,  as  does  its  rate  of  change  and  its  other  time 
derivatives.  In  a  computerized  system,  the  rate  of  data  input 
as  well  as  processing  time  are  both  limited. 

In  terms  of  the  vector  U,  these  constraints  may  be 
expressed  by  sin  inequality  governing  each  component.  Thus: 

U j  ~  K j »  (2) 

where:  K.  is  a  constant  for  each  j  =  1,  ...»  n. 

When  a  plant  is  controlled  automatically,  or  when  a  model 
representing  the  operator's  control  policies  has  been  developed. 
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the  control  vector  U  may  be  expressed  entirely  as  a  function  of 
the  system  state.  That  is, 

Uj  *  U  (X)  for  j  =  1 ,  . . . ,  n  (3) 

Substitution  of  equation  (3)  into  equation  (1)  yields,  then,  a 
closed  set  of  equations  representing  the  dynamics  of  the  plant. 

Let  us  now  assume  that,  for  a  given  resource,  we  can 
represent  the  rate  at  which  that  resource  is  being  used  by  a 
scalar  function  F  of  the  state  and  control  variables.  By  integrating 
this  function  over  the  time  during  which  the  plant  is  in  operation 
(or  over  any  shorter  interval),  we  obtain  a  summary  measure 
(SUMPM)  of  the  plant's  performance .  That  is: 

T 

SUMPM  =JF  (X,U)  dt. 

fco  (4) 

In  order  to  insure  that  the  resources  used  accumulate,  and  thus 
to  insure  that  plant  performance  at  present  cannot  cancel  measure¬ 
ment  of  resources  used  earlier  -  i.e.,  to  insure  that  correct 
contrdl  of  the  plant  is  a  function  only  of  its  present  state  -  the 
function  F  is  defined  such  that  it  is  either  zero  or  positive,  but 
never  negative.  Thus: 

F  (X,U)=*  0.  (5) 

We  will  now  state  below  without  further  development  the 
definition  of  the  System  Performance  Measure,  SYSPM. 

This  definition,  which  relates  the  SYSPM  to  the  SUMPM,  can  be 
used  to  measure  the  significance  of  deviations  from  correct  control 
(as  it  is  here)  as  well  as  for  such  other  purposes  as  synthesizing 
optimal  control  laws.  For  a  more  complete  and  rigorous  develop¬ 
ment,  the  reader  is  referred  to  Connelly,  et  al.  (1969), 

Connelly,  et  al.  (1977),  and  the  proofs  given  in  the  Appendix. 

We  can  write  the  SYSPM  ih  either  of  two  forms: 

SYSPM  =  $  (X)  +  F(X,U)  (6a) 
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(6b) 


SYS  PM  =5  j-^-X  +  F(X,U) 
i=1  3Xi  1 

In  this  definition.  Theta  (0)  is  a  function  of  the  state 
variables  only,  and  is  termed  the  "resources-to-go"  function.  It 
is  a  positive  definite  function  which  gives,  at  any  time,  depending 
on  the  state  of  the  plant  alone,  the  resources  required  to  complete 
the  task  assuming  that  reference-task  performance  is  used  from 
t±ie  present  state  to  the  end  of  the  task.  Referring  to  Figure  1, 
where  the  SUMPM  has  been  plotted  against  time,  we  see 
that  theta  equals  0  at  the  end  of  the  plant  trajectory  (at  time  T), 
and  that,  at  the  beginning  of  the  trajectory  (at  time  t  )  it  equals 
the  final  value  of  the  SUMPM.  The  time  derivative^  of  theta,  0, 
is  the  rate  at  which  "resources-to-go”  are  required. 

Equation  (6a)  allows  us  to  examine  the  relationship  between 
F  and  0.  F,  since  it  is  a  function  of  the  control  vector  U, 
can  be  thought  of  as  the  rate  at  which  resources  are  expended 
whether  or  not  the  system  is  being  controlled  for  optimal  (or 
reference)  performance.  On  the  other  hand,  0,  since  it  is  a 
function  of  the  state  vector  only,  is  an  indicator  of  the  "distance- 
to-go"  to  complete  the  task.  When  optimal  (or  reference)  control 
is  applied,  the  rate  at  which  resources  are  expended  is  exactly 
being  offset  by  the  rate  at  which  resources  are  required,  and 
value  of  0  is  equal  to  -F.  Thus,  when  optimal  (or  reference) 
control  is  applied,  the  value  of  the  SYSPM  equals  0. 

Equation  (6b)  merely  emphasizes  the  state  nature  of 
the  SYSPM,  by  allowing  it  to  be  written  entirely  in  terms  of  the 
state  vector  X.  Thus,  by  means  of  equations  (1)  and  (3)  we  obtain: 

N  -  a 

SYSPM  =  -S  2-f.  (X,U[X])  +  F(X,U)  (7) 

-j  *  xi  1 

We  have  seen  from  our  discussion  of  equation  (6a)  that  the 
SYSPM  equals  0  whenever  reference  control  is  applied.  But  we 
may  also  interpret  this  equation  as  a  measure  of  the  significance 
of  a  deviation  at  any  time  from  reference  control.  When  system 
control  errors  occur  which  do  not  particularly  affect  mission 
performance,  the  value  of  the  SYSPM  will  be  very  small, 
almost  zero.  Conversely,  any  large  increase  in  the  value 
of  the  SYSPM  may  be  taken  as  a  direct  indicator  that 
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significant  control  errors  are  being  made.  Integrating  the  SYSPM 
yields,  over  the  time  interval  chosen,  a  measure  of  the  quality  of 
the  system  control. 

At  any  time,  it  is  always  possible  for  the  pilot  (operator) 
to  set  the  SYSPM  equal  to  0  by  simply  selecting  control  values 
which  cause  $  to  be  equal  to  -F.  The  function  9  insures  that 
the  correct  (or  reference)  control  results  in  performance  that 
minimizes  the  integral  of  F.  Thus,  the  measure  function  removes 
the  dynamic  lags  of  the  controlled  system  from  the  performance 
assessment.  Naturally,  the  response  of  any  system  to  actual 
control  changes  does  in  fact  depend  on  the  system’s  dynamics. 

The  SYSPM  simply  removes  this  dependency  from  performance 
measurement,  and  thus  does  not  penalize  system  operators 
because  the  system  itself  cannot  change  instantaneously. 

The  notion  of  removing  the  dynamics  of  the  controlled 
system  (in  this  case  the  aircraft  dynamics)  is  important  because 
it  means  that  at  each  instant  of  time  it  is  possible  for  the  pilot 
(operator)  to  provide  a  correct  input  -  an  input  that  will  be 
scored  by  the  performance  measure  as  correct.  If  we  account 
for  the  0.2  to  0.5  second  lag  in  human  processing  times,  then 
we  can  say  that  performance  during  successive  .5  second  time 
intervals  can  be  considered  as  independent.  Thus  we  argue  that  in  a 
second,  two  or  more  independent  tests  of  operator  performance  are 
administrated.  Actual  performances  during  successive  intervals 
may  be  correlated  due  to  a  correlation  of  the  operator’s  control 
policy  but  this  correlation  is  due  to  the  operator  rather  than  the 
other  system  dynamics. 

The  result  of  this  measurement  system  permits  evaluation 
of  performance  as  a  function  of  system  states  or  conditions  that 
would  otherwise  have  to  be  aggregated  over  the  total  task  in 
a  summary  measure.  This  permits  evaluation  of  the  effect  on 
performance  of  factors  that  influence  performance  on  only  a 
portion  of  the  total  task,  or  whose  influence  is  not  uniform  over 
the  task,  or  are  error  related  factors  (such  as  size  of  the  error). 

In  our  discussion  above,  9  was  defined  in  terms  of  F,  given 
Optimal  control  over  the  system.  In  empirical  development  of  a 
system  performance  measure,  the  optimal  control  law  is  generally  not 
known  in  explicit  form,  and  may  never  be  known.  In  such  cases,  data 
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from  various  levels  of  performance  (levels  to  be  determined 
by  the  SUMPM)  are  collected,  and  an  approximation  to  $  is 
developed.  This  empirically  derived  "resources-to-go"  function 
might  be  termed  the  "resources-to-go-given-reference- 
demonstrations~of-performance,,  function . 


METHOD  OF  APPROACH 


A  brief  presentation  of  the  theory  of  performance 
measurement  used  here  is  given  in  a  previous  section.  This 
theory  applies  directly  to  situations  where  the  factors  limiting 
the  performance,  equations  of  the  controlled  system  (aircraft), 
and  performance  criteria  are  known.  A  proof  of  the  necessary 
and  sufficient  conditions  for  optimality  is  given  in  Appendix  A. 

The  method  employed  here  used  a  somewhat  different 
approach  which  can  be  applied  where  performance  demonstrations 
are  available  but  the  demonstrations  are  not  scored  or  ordered. 

The  method  uses  demonstration  data  to  develop  a  model  of  the 
system  (aircraft)  dynamics  and  performance  criteria.  To 
accomplish  this,  the  methods  given  in  the  appendix  have  been 
adapted  to  a  procedure  whereby  the  demonstration  data  is  analyzed 
to  develop  a  set  of  approximate  aircraft  equations  and  representative 
pilot  control  policies.  These  equations  and  policies  are  used  to 
construct  a  measure  which  will  indicate  the  convergence  of  at 
least  some  performance  demonstrations.  The  measure  is  then 
tested  and  applied  to  all  performance  demonstrations  so  that 
all  demonstrations  are  scored  with  a  consistent  measure. 

The  specific  approach  was  to  construct  a  second  order 
performance  model  which  measures  performance  according  to 
how  well  the  pilot  controls  the  second  error  derivative  given 
the  error  and  its  first  derivative.  For  instance,  in  longitudinal  glide 
path  control,  the  second  derivative  of  the  glide  path  error  is 
evaluated  as  a  function  of  the  glide  path  error  and  its  first 
derivative . 

In  qeneral,  a  model  of  any  order  desired  for  a  specific 
problem  c«n  be  constructed.  If  it  is  knowr\  for  instance,  that  the 
pilot's  (operator's)  controls  directly  affect  the  first  derivative  of 
the  error,  then  a  first  order  model  should  be  used.  The  objective 
is  to  use  a  model  of  an  order  that  permits  evaluation  of  a 
derivative  that  can  be  or  is  rapidly  adjusted  by  the  pilot 
(operator).  Tine  ideal  model  would  permit  evaluation  of  the 
control  element  (throttle,  control  stick)  as  a  function  of  system 
state.  However  as  will  be  seen  it  is  desirable  for  computational 
simplicity  to  evaluate  performance  of  a  variable  which  is  dynamically 
"close"  to  the  pilot's  control  elements,  i.e.,  a  variable  that  can  be 
rapidly  modified  by  the  pilot. 
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The  subsequent  paragraphs  provided  a  description  of 
the  method  of :  formulating  the  aircraft  model  and  solving  for 
the  summary  performance  measure  coefficients.  Also,  the 
method  for  developing  the  convergence  indicator  and  system 
performance  measure  (SYSPM)  is  also  described.  Note  that 
the  convergence  indicator  is  the  cost-to-gv  function  ($)  described 
in  the  subsequent  paragraph.  After  the  presentation  of  the 
theory,  the  results  of  the  data  analysis  as  well  as  results  of 
the  comparative  analysis  task  (T ask  C  of  the  program)  are 
presented . 

Specific  Method  of  Approach 
Take  the  set  of  plant  equations  as: 

X1  =  X2  (8) 

X2  =  _aX1  "bX2  +  U  (9) 

where:  X^ ,  X2  are  state  variables  and  U  is  a  control  variable. 

The  index  function  is  taken  as: 
t. 

I=  /  F<?VX2'U>dt  (10) 

o 

where: 

F  =  A1  X^  +  A2  X1  X2  +  A3  X22  +  A4  U2  (11) 

and  F  >  0  except  at  the  origin.  Conditions  insuring  that  F  >  0 
are  that  A  ,  A  >  0  and  the  quadratic  has  imaginary  roots.  This 
requires  that  A*  <  4A1A3«  (12) 


The  theory  of  optimality  says  that  the  control  function 
U  which  minimizes  the  function  I  must  minimize  6  at  each 
point  in  the  state  space  where: 


3  9  * 

d>  =  2- —  X.  +  F 
9  .  3x.  i 

i  i 


(13) 
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and  $  is  a  function  of  the  state  variables  to  be  determined.  Thus, 


<t>  =  x_  +  — —  (-ax.  -  bx  +  U) 
^x^  2  jx2  1  2  ' 


1 

a  0  ZB 


2  12 
U,  ^ 

1 


» _ 

3  2 


+  A  U2 

4 

(14) 

>  0 

(15) 

a  * 

The  first  derivative  ^  y  set  equal  to  zero  determines  the 
value  of  U  for  whicba  takes  on  an  extreme  value.  Since  the 
second  derivative  |  ^  is  positive,  that  extreme  value  is  the 
minimum  value  of  +■.  Thus  the  desired  value  of  U  is: 


U*  -  ^  -  d  9 

2A4  3x2 


(16) 


The  function  8  (X^ ,  X  )  is  determined  such  that  the  minimum 
value  of  0  is  zero.  If  U  =  U*,  ^  takes  on  its  minimum  value 
fOr  all  t  (t  >t0  )  (where  t  is  die  initial  time)  and  it  is  known 
(Elgerd,1967)  that  $  has  the  form 


=  8(X1,  Xo)  =  B  X2  +  B  X  X  +  B  X  2 
1  2'  1  2  12  3  2 


|_1 „ 2Bl  X,  +b2  xa 
■  B2  Xt  +  2  B3  X2 


07) 

(18) 

(19) 


Substitution  of  the  partials  into  the  equation  for  £  and  setting^ 
=  0  yields 


(20) 


B1  =  B1  <V  A2’  A3'  V 


Brt  =  B  (A  ,  A  ,  A  ,  A  J 
2  2  v  1  2*  3  4J 


B  =  B  (A  ,  A  ,  A  ,  A  J 
3  3  1  2  3  4 


also  note  that 


U*  =  — J-  =  -  rr"  (B-  X  +  2  B  Xrt) 

2A^  dX_  2A,  v  2  1  3  2 

4  2  4 


and  the  original  set  of  differential  equations  become 


X1  =X2 


X0  =  -aX,  -bX0-“-  (B_  X  +  2  Bn  Xrt) 
2  1  2  2A  v  2  1  3  2 

4 


which  is  a  linear  system. 


It  must  be  noted  that  the  values  of  (A  ,  A  ,  A  ,  A  )  must  be 
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taken  to  insure  that  F  and  9  are  positive  definite  functions. 


Next,  the  fljnction  selected  for  6  must  be  entered  into  $ 
along  with  the  equation  for  U,  so  that  the  values  for  B-j ,  Bq,  B3, 
and  A-j.,  Aq,  A3,  A4  can  b^ determined  which  set  $  to  0.  Since 
a,  b  along  with  the  ratios  gjf—  and  — —  are  determined  from 
flight  data,  values  of  Bg,  Bg,  A^,  a  and  b  are  used  to  determine 
values  for  A1 ,  A2,  Ag  and  B1  as  follows: 


+  B2  /2A4 


2aBo  +  bB_  +  A„ 
J  2  4 


=  2B_b 


+  -Pa — e 


1  U!  ~  i!V-'  ' 


Tests  are  run  to  ensure  that  the  following  conditions  are  satisfied; 


A,  >  0,  A4  >  0 

a22  <  *Va 


(29) 

(30) 


and 


B,  >  0 

B22  <  4B1  S3 


(31) 

(32) 


which  ensures  that  the  quadratic  forms  have  imaginary  roots. 
Conditions  31 ,  32  are  easily  satisfied  since  B  can  be  selected 
to  be  any  value  desired.  But  then  condition  30  must  be  checked 
to  ensure  that  it  is  satisfied. 


The  procedure  for  establishing  the  measure  for  second  order 
control  is  to; 

1.  Identify  the  aircraft  parameters  a,  b. 

2.  Identity  the  control  equation  parameters  for 
flight  demonstrations  selected  as  representative 
of  superior  performance. 


Note  that  Step  2  uses  equation  16,  i.e.. 


U*  = 


1 

2A 

**4 


(B2  X  +  2B3  X2) 


to  establish  the  values  for  the  two  ratios: 


(33) 
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3.  Values  A1 ,  Ag,  A3,  and  Bi  are  selected  to  satisfy 
conditions  28  and  30,  31 .  Then  the  performance 
measure  4  is  determined  from  Equation  14. 
According  to  the  theory  given  in  the  appendix, 
the  measure  is  equal  to 


<t>  =  Aa  (U  -  uV 


(34) 


where 


U*  = 


— -  (B  X  +  2BX). 

2A.  v  2  1  3  2y 


(35) 


Performance  Measure  Parameters 

In  order  to  establish  the  <fynamics  of  the  system,  i.e., 
obtain  coefficient  values  for  Equation  9,  and  to  obtain  initial  data 
for  reference  control  functions,  four  regression  analyses  were 
used.  One  regression  was  used  to  oredict  the  second  derivative  (with 
respect  to  time)  of  the  glide  slope  error  as  a  function  of  glide 
slope  error,  its  first  derivative*  and  the  throttle  (control).  A 
similar  regression  used  the  second  derivative  of  angle  of  attack 
as  the  dependent  variable  and  the  angle  of  attack  error,  its  first 
derivative,  and  elevator  as  the  independent  variables.  In  equation 
form  the  regression  equations  for  glide  path  and  angle  of  attack 
control  are  (respectively): 


2  ^ 

d  GPE 

2 

dt 


*  K  +  K, 


d  GPE 
dt 


K3  GPE  + 


K 


(36) 


/\ 

2 

d  a 


dt 


■  K5+K6 


da 

dt 


+  K?a 


V. 


@7) 


where 


GPE  is  the  glide  path  error 

T  is  the  throttle  position 

a  is  the  angle  of  attack 

S  is  the  elevator  position 

© 

"/s"  indicates  a  predicted  (dependent)  value 
for  the  indicated  variable 

Samples  of  flight  data  were  analyzed  using  the  regression 
analysis  with  the  results  shown  in  Tables  1  and  2.  Table  1  is 
for  the  longitudinal  glide  path  analysis.  One  observation  from 
the  Table  1  data,  as  shown  in  the  right  most  column,  is  that 
eventhough  there  is  considerable  variation  in  the  constant  «•) 
and  the  throttle  control  coefficient  their  ratio  is  relatively 
stable.  This  ratio  was  interpreted  to  be.  the  throttle  offset 
value  such  that  if  the  throttle  equals  the  offset  value  and  the 
*  glide  path  error  and  error  rate  are  zero,  the  glide  path 
acceleration  is  approximately  zero.  Thus  the  equation  was 
taken  as: 


d2  GPE 
dt2 


=  K 


d  GPE 
dt 


+  K  GPE  +  K  (T  - 
3  4 


% 

K 


-)  (38) 


Referring  to  Table  2,  a  sizeable  constant  appeared  in  the 
angle  of  attack  equation  which  except  for  one  large  value  (file  26) 
appeared  to  equal  the  desired  a  value  plus  an  offset  for  the 
elevator.  Thus,  the  angle  of  attack  equation  was  taken  as: 


K 


da 

dt 


+  K„  («  -  15).  +  K 
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<*•- 


w 


(39) 


Considerable  variations  in  coefficient  values  were  found 
as  shown  in  Tables  1  and  2.  From  a  control  theory  viewpoint  the 
control  feedback  coefficient  for  error  rate  should  be  negative  in 
order  to  have  a  stable  system  without  additional  damping  from 
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TABLE  1  ..  REGRESSION  TABLE:  GLIDE  PATH  DEVIATION 


%  Error 

Variance  Rate 


Error 


Throttle 

Control 


Constant 

Control 


File  Explained  Coefficient  Coefficient  Coefficient  Constant  Coefficient 

K2  K3  K4  Kj  K1/K4 


.029 


.049 


-.00458 


-.0088  -.00249 


-.0043  -.008 


.0202  - .0141 


.0677  -.0026 


-.0198  -.00763 


-.00662 


.0246  t . 00887 


-.0066  -.0038 


-.0059 


-.00983 


-4.375 


-12.7 


-7.998 


-  .568 


-6.138 


-3.559 


-2.886 


-5.221 


-5.301 


-3.096 


-2.884 


-2.821  .644 


-8.096  .637 


-5.234  .654 


-  .435  .765 


-4.014  .653 


-2.418  .679 


-1.913  .662 


-3.519  .674 


-3.496  .659 


-2.055  .662 


-1.851  .641 
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TABLE  2.  REGRESSION  TABLE:  ANGLE  OF  ATTACK 


%  Error  Control  Constant 

Variance  Rate  Error  Coefficient  Error 


File 

Explained  C 

oefficient 

K6 

Coefficient  (El^ator) 

Constant 

K5 

12 

40 

.517 

-.205 

1 .314 

2.808 

13.79 

13 

26 

.497 

-.223 

3.619 

3.146 

13.34 

23 

22 

.647 

-.275 

3.387 

3.862 

12.88 

24 

38 

.462 

-.223 

4.123 

3.08 

13.1 

25 

48 

.414 

-.216 

3.893 

3.082 

5.8 

26 

39 

.472 

-.204 

2.431 

2.935 

354. 

35 

35 

.599 

-.243 

4.164 

3.505 

11.21 

36 

41 

.387 

-.201 

3.037 

2.86 

11.89 

37 

26 

.429 

-.098 

-5.207 

1  .43 

18.19 

38 

45 

.565 

-.253 

.056 

3.681 

21  .25 
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the  pilot’s  control  policy.  The  inconsistent  sign  of  the  error 
rate  coefficient  for  flide  path  control  was  interpreted  as  due 
to  effects  of  both  wind  and  oscillatory  control  of  the  aircraft. 

Also,  the  positive  value  for  the  error  rate  coefficient  for  angle 
of  attack  control  was  attributed  to  similar  causes.  It  should 
be  noted  that  angle  of  attack  control  was  typically  oscillatory 
of  all  flights. 

Coefficient  values  representing  an  approximate  second 
order  model  of  the  aircraft  were  selected  as  typical  for  the 
regression  table  values.  For  both  the  glide  path  and  angle  of 
attack  controls  a  stable  (i.e.,  negative)  coefficient  value  was 
selected.  However,  as  will  be  seen  in  the  subsequent  analysis, 
a  reference  pilot  control  function  must  be  selected  so  that  the 
resulting  system  is  stable.  As  a  result,  almost  any  error  rate 
coefficient  for  the  aircraft  can  be  selected  since  it  will  be 
compensated  by  reference  pilot  control  function.  In  contrast, 
the  error  coefficient  which  establishes  the  system  natural 
frequency  is  modified  only  moderately  by  the  reference  pilot 
control  function. 

As  a  first  step  in  establishing  a  reference  control  function 
for  glide  path  and  «  control  via  throttle  and  elevator  respecitvely, 
two  additional  regression  analyses  were  run.  These  regressions 
used  throttle  and  elevator  positions  as  the  dependent  variables  and 
the  respective  errors  and  error  rates  as  the  independent  variables. 
Results  of  these  regressions  are  shown  in  Tables  3  and  4. 

Referring  to  Table  3,  the  constant  of  approximately  .65  is  equal 
to  that  found  in  Table  1  and  was  used  as  the  offset  for  the  throttle. 
No  offset  was  used  for  the  elevator. 

Coefficient  values  for  the  two  reference  control  functions 
are  given  in  Table  5.  Values  for  a  and  b  were  selected  as 
representative  of  those  values  obtained  in  the  regression  analyses. 
Values  for  flight  (file)  25  were  used  except  that  the  error  rate 
coefficient  was  given  a  negative  sign.  Using  the  procedure  outlined 
previously,  a  value  for  B2/2A4  is  determined  from  Table  5  along 
with  an  initial  value  for  B3/2A4.  However  it  was  necessary  in 
both  cases  to  increase  B3  (i.e.,  to  provide  more  damping)  for 
the  reference  control  in  order  to  satisfy  the  realizability  conditions 
29,  30,  31 ,  and  32.  Thus  the  flight  selected  as  a  reference  (25) 
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TABLE  3.  REGRESSION  TABLE:  THROTTLE  CONTROL 


%  Error 


File 

Variance 

Explained 

Rate 

Coefficient 

Error 

Coefficient 

Constant 

11 

39 

.00136 

.0003 

-.64 

12 

29 

-.00204 

.00013 

-.64 

13 

75 

-.0035 

.00055 

-.64 

23 

78 

.0011 

.00408 

-.669 

24 

56 

.0068 

.00132 

-.669 

25 

12 

.00152 

.000321 

-.648 

26 

20 

.0033 

! 00025 

-*652 

35 

34 

.00364 

.00064 

-.641 

36 

45 

-.00337 

.000434 

-.654 

37 

66 

.00509 

.00256 

-.632 

38 

24 

.0084 

.00291 

-*653 

TABLE  4 


REGRESSION  TABLE:  ELEVATOR  CONTROL 


File 

% 

Variance 

Explained 

Error 

Rate 

Coefficient 

Error 

Coefficient 

Constant 

Constant 

Error 

Coefficient 

12 

20 

.0286 

.029 

2.808 

2.13 

13 

39 

.0102 

.0266 

3.146 

.869 

23 

37 

.0082 

.0239 

3.862 

1.14 

24 

18 

-.0096 

.0167 

3.08 

.911 

25 

5 

.0065 

.0058 

3.082 

.791 

26 

1 

.0064 

-.00014 

2.935 

1  .2 

35 

17 

.0069 

.0126 

3.505 

.841 

36 

21 

-.0046 

.0158 

2.86 

.943 

37 

12 

.0377 

-.0094 

1  .43 

.275 

38 

24 

.0581 

-.0064 

3.681 

17.24 
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TAELE  5.  COEFFICIENT  VALUES  FOR  ThE  REFERENCE 
CONTROL  FUNCTIONS 


Glide  Path  Channel 
-2 

-  .72  x  10 

-  .2  x  10"1 

-5 

.8x1. 

.59  x  10~4 
.208  x  10“2 
1  .0 

.292  x  10~3 
. 1 1 2  x  1 0"2 
.4x10  1 


Alpha  Channel 
-  .216 
-.414 
.325  x  10_1 
.153  x  10"1 
.728  x  10~2 
1  .0 

.22  x  10“ ’ 
.454  xIO"1 
.1 
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did  not  exhibit  a  stable  control  as  measured  by  the  form  of  the 
function  (Equation  17)  selected.  Another  functional  form  may 
permit  accepting  that  flight  control  as  stable;  but,  for  computational 
simplicity  the  form  of  Equation  17  was  used  and  the  value  of  B3 
adjusted  to  satisfy  the  realizability  conditions. 
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ANALYSIS 


Data 

The  data  on  carrier  landings  are  available  on  9-track 
magnetic  tape  and  consisted  of  flights  by  four  subjects  each 
performing  on  12  flights.  Each  flight  was  performed  on  a 
particular  combination  of  glide  path  error  display,  and  day/ 
night  combination.  Each  display/time  combination  is  defined 
as  a  treatment  and  given  a  number  as  shown  in  Table  6.  Thus 
Treatment  1  uses  a  conventional  display  and  day  flights  while 
Treatment  2  uses  a  command  display  and  day  flights.  Table  7 
shows  the  treatment  design  applied  to  each  subject  trial  in  sequence. 
The  first  subject  trial  used  Treatment  1  (conventional  display  and 
day  flights).  Trial  2  used  Treatment  2  and  so  forth.  It  is  seen 
that  each  treatment  is  given  to  each  subject  exactly  three  times  in 
a  randomized  order. 


TABLE  6.  TREATMENT  DEFINITIONS 


Treatment  Number 

Decent  Rate  Cueing 

Time 

1 

Conventional 

Day 

2 

Command 

Day 

3 

Conventional 

Night 

4 

Command 

Night 

TABLE  7.  TREATMENTS  FOR  SUBJECT  TRIALS 
Subject  Trial  T reatments 


1  1 

2  2 

3  4 

4  3 

5  3 

6  4 

7  2 

8  1 

9  1 

10  2 

11  4 

12  3 
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Data  Analysis  Procedure 


The  analysis  procedure  is  given  in  schematic  form  in 
Table  8.  The  flight  data  which  consists  of  samples  of  22  variables,  each 
sampled  30  times  a  second,  was  analyzed  by  taking  every  10th 
example.  Since  the  performance  measurement  algorithm  permits 
an  immediate  evaluation  of  a  sample,  a  score  was  developed 
for  each  (10th)  sample  (hereafter  called  "sample”)  for  both  glide 
path  control  and  angle  of  attack  control.  The  scores  were 
classified  according  to  three  factors.  One  factor  is  the  treatment: 
display  and  time  of  day  combination  described  previously. 

Another  factor  is  the  horizontal  distance  from  the  landing  deck. 

This  is  divided  into  nine  regions  each  of  which  is  1 ,000  feet 
in  length  as  shown  in  Table  9. 

The  third  factor  is  an  error/error  rate  cell  as  shown 
in  Figure  2.  To  simplify  the  calculations  two  categories  of 
error  and  error  rate  each  were  defined.  Performance  scores 
were  categorized  depending  on  whether  or  not  the  absolute  value 
of  glide  path  error  was  greater  than  or  less  than  15  feet. 

Similarity,  performance  scores  were  categorized  as  a  function 
of  the  absolute  value  of  the  glide  path  error  rate  being  greater 
or  less  than  2  feet/second.  The  combination  of  error/error  rate 
categories  leads  to  four  cells  as  defined  in  Figure  2. 

Return  now  to  the  description  of  the  computational 
system  as  shown  in  Table  8.  Each  data  sample  was  analyzed 
to  produce  a  score  value.  The  scores  were  grouped  according 
to  the  three  categories  just  described  and  collected  for  each  trial. 
Recalling  that  an  individual  flight  (also  referred  to  as  a  trial)  is 
associated  with  only  one  treatment,  the  data  for  a  trial  represents 
the  performance  of  one  subject  performing  with  one  treatment 
and  the  data  is  categorized  by  the  error/error  rate  cell  and 
range  sector.  It  is  possible  of  course  to  aggregate  this  data  in 
many  different  ways,  such  as  combining  performance  of  one 
subject  according  to  overall  treatment  or  by  all  subjects  on  one 
treatment  or  any  combination  of  the  categories  desired. 
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TABLE 


8.  COMPUTATION  SYSTEM 


Scores  for  each  subject** 
for  each  combination  of: 

-  sector 

-  error/error  rate  cell 

Subject  Trial*  -  treatment _ 


1 


1 

2 


2 

3 


3 

4 


4 


1 


12 

13 


24 

25 


36 

37 


48 


Scores  for  Subject  #1 


} 

) 

} 


Scores  for  Subject  #2 


l  *** 


Scores  for  Subject  #3 


Scores  for  Subject  #4 


*Trial  analysis  for  each  trial 

**Analysis  for  each  combination 

-  sector  (9) 

-  error/error  rate  cells  (4) 

-  treatments  (4) 

***Combined  scores  for  each  combination 

-  sector 

-  error/error  rate  cell 

-  treatment 
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TABLE  9.  SECTOR  DEFINITIONS 


Sector  Number 
1 
2 

3 

4 

5 

6 

7 

8 


Horizontal  Distance  (in  ft) 
For  Landing  Deck 
Center  Line  (x) 


x  ^  1000  ft 


1000  ft  z  x  ^  2000  ft 

2000  ft  ^  x  3000  ft 

3000  ft  ^  x  4000  ft 

4000  ft  x  5000  ft 

5000  ft  4  x  ^  6000  ft 

6000  ft  z  x  ^  7000  ft 

7000  ft  x  ^  8000  ft 


9 


8000  ft  ^  x 


Error  Rate 


<  2  ft/sec  >  2  ft/sec 


<  15  ft 


Error 


>15  ft 


1 

4 

2 

t 

3 

Figure  2.  Definition  of  glide  path  error  and 
error  rate  cells. 


Recall  that  the  score  is  developed  as  the  control  error 
squared  Equation  (34).  It  is  recognized  that  even  though  the 
control  error  may  be  normally  distributed  (and  this  is  an 
assumption),  the  squaring  operation  folds  the  negative  control 
error  values  onto  positive  values  so  that  the  distribution 
becomes  bounded  on  the  lower  end.  Thus,  the  distribution 
of  the  score  values  cannot  be  normal,  but  must  be  represented 
by  a  distribution  that  is  bounded  on  the  lower  and  open  on  the  upper 
end.  As  a  result,  the  standard  deviation  of  sample  values  is 
likely  to  be  equal  to  or  perhaps  even  exceed  the  mean  score. 

In  addition  to  the  development  of  sample  scores,  a 
set  of  analyses  were  performed  to  evaluate  average  scores  for 
individuals  and  to  examine  the  effective  treatment  over  the  average 
scores  for  all  subjects.  This  procedure  was  to  develop  average 
scores  for  each  individual  subject  over  the  12  trials  he  performed 
and  to  categorize  those  scores  according  to  the  three  factors 
described  previously.  Note  that  the  number  of  categories  is 
9(sector)  x  4(error/error  rate  cells)  x  4(treatments)  =  144 
categories . 
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Analysis  Steps 


Analysis  Step  1 . 

The  purpose  of  Step  1  is  to  compute  the  mean,  variance 
and  other  parameters  of  the  performance  measure,  and  to  summarize 
performance  in  sectors  within  each  data  file  (flight).  This  analysis 
was  performed  for  each  flight  using  every  10th  data  sample  which 
provides  a  sample  interval  of  0.333  seconds.  In  addition  to  the 
data  documentation,  the  distribution  of  data  samples  is  to  be 
determined. 

It  is  not  practical  or  desirable  to  print  the  results  of  all 
48  flights.  Instead  Figure  31  presents  the  results  of  Flight  12  which 
is  the  last  flight  for  the  first  subject.  The  total  number  of 
examples  for  this  flight  Is  1,699  samples.  Typically  all  flights 
had  approximately  that  total  amount  of  samples;  however,  sampling 
every  10th  sample  provided  approximately  170  samples  per  flight. 
Referring  to  the  figure,  the  term  Jcode  *  1  indicates  glide  path 
control  while  Jcode  *  2  indicates  angle  of  attack  control.  Scores 
are  assigned  to  one  of  nine  sectors  each  of  which  correspond  to 
a  1 , 000  foot  (horizontal)  segment  where  Sector  1  is  closest  to 
the  landing  deck  (l.e.,  zero  to  1,000  foot). 


The  data  presented  at  the  top  of  the  figure  are  the  mean 
PM,  the  PM  variance,  as  well  as,  the  max  and  min  PM  for  each 
sector.  Also  presented  is  the  max  rate  of  change  of  9  which 
is  a  measure  of  the  rate  of  convergence  (or  divergence)  of  the 
flight  trajectory  to  the  glide  path.  If  the  flight  trajectory  is 
convergent  as  measured  by  the  resource-to-go  function  (  $  ) 
then  the  maximum  value  of  9  would  be  negative.  But  the  oscillatory 
response  of  the  aircraft  provides  some  positive  values  for  the 
rate  of  change  of  9. 

Note  that  a  large  penalty  score  is  given  in  Sector  9  where 
the  penalty  is  considerably  greater  then  that  of  other  sectors.  A 
"start  up"  error  in  calculating  derivatives  caused  the  large  penalty 
score  -  it  is  not  due  to  pilot  performance. 
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Ifft  FILfc  NLtttR  *  12  (Right  12  for  Subject  1) 

SrtUt  khlfc  *  10  (Every  10th  data  point,  i.e.,  3/sec) 

10’lft  SttHkS  *  16%  (Total  number  of  data  points  for  this  trial) 

JtUJt*  i  (Glide  path) 

SEtKfc  n  m  m  m  ms  i«: 

ftffl  PH  Pf!  VHtlh  L.:.T 

1  1?  0.14816*68  6.M6SE-42  8.31&4E+88  8.5153Hil  6.160**61 

2  10  0.  1573t-61  6.181%-03  0.4656E-6i  0.24056-63  6.138**66 

3  20  0.5241fc-01  6.1317E-02  ‘MtiSfM  0.4433E-63  6.1i;%n*.i 

4  10  0.584tt-61  0. 1022-02  8.1&:£*60  8.1C4E-91  6.*4fi*66 

5  10  0. 1240E-01  0.6872E-84  0.2442-01  0.3C80E-W  0,4*446*66 

6  10  0.1632-01  0.2470E-03  0.4404E-01  6..122€-65  6.&^:t*66 

7  20  0.3012-01  0.5322-03  0; 6152-01  0.15&7E-O3  6,24e8c*06 

8  10  0.12216*68  0.12736-81  6.3262*00  9.57186-83  y. 11356*6! 

0  1?  0.63/76*01  0.6142*03.  6.1052*03.  0.2342-01  8. 02406*04 

101ft  0.6022*60  0.6544£*02  0.1052*03  8.1224M5  8.  ^06*64 


SfcCKfc  CtJOE  N 

1  1  1 

1  3  1 

1  4  15 

2  2  7 

2  3  6 

2  4  6 

3  1  3 

3  4  17 

4  2  8 

4  3  1 

4  4  10 

5  2  1 

5  3  5 

5  4  13 

6  2  7 

6  3  12 

7  2  15 

7  3  5 

8  2  2 

S  3  17 

1  1  2 

0  3  8 

0  4  7 


Kffl  »H  Si 
Ft! 

0.12S056  0.01174% 
0.0515330  >6.0026557 
0.1571652  0.624790* 
6.0160460 

0.0160682  0.6002563. 
6.0140413  0.0001072 
0.0450426  8.0021107 
0.0535516  0.602%.?;' 
0.0254712  0.00%483 
6.0444087  0*0010301 
8.0861340  0.0674102 
0.00660%  0.0006%.% 
0.6057102  0.0006327 
0.0152058  6.0902340 
8.0357202  8.'6812'it? 
0.8050741  6.ttfo57 
8.0500316  0.«025?4fe 
6.8038461  0.0008:48 
0.6448124  8.862t&32 
0.1311060  b.8172110 
0.0265548  0.089.95.2 
8.2731451  0.87461^3 
15.1678385  230.86307% 


Glide  Path  Control 


'igure  3 .  File  data  analysis  (file  12).. 
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The  lower  part  of  the  figure  gives  a  result  of  a  further 
decomposition  where  each  data  sample  score  is  assigned  to  a 
sector  and  error/error  rate  category.  The  score  shown  is 
the  mean  score  for  the  indicated  sector,  error/error  rate 
combination  along  with  the  number  of  data  samples  in  that 
category.  Also  note  that  the  large  penalty,  identified  above, 
which  occurs  in  Sector  9  can  now  be  seen  to  occur  when  the 
error/error  rate  code  is  4  -  which  corresponds  to  a  small 
glide  path  error  and  a  large  glide  path  "error"  rate,  i.e., 
the  computational  startup  error  has  been  conveniently  collected 
in  one  cell. 

Consider  now  the  distribution  of  scores  for  each  data 
sample.  For  a  first  analysis  it  is  assumed  that  the  sampled 
control  errors  are  uncorrelated  and  are  normally  distributed 
with  a  density  functions 


f  (X) 
x' 


-x2/2* 

e 


2 


where  2 

x  is  the  error  with  a  mean  of  zero  and  *  is  the  variance 
of  the  error  distribution. 

The  performance  measure  employs  the  transformation: 

PM  =  a  X2 

so  that  the  measure  scores  are  not  distributed  normally  but  instead 
the  distribution  is  folded  where  the  performance  measure  takes  on 
only  positive  values.  Papoulis  (1365)  gives  the  desired  distribution 
density  Punction  as: 


WPM>  * 


U(PM) 


-  PM/2  a  <7  ‘ 


\2raPM 
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where  U(PM)  is  a  unit  step  function  such  that 

U(PM)  a  0  for  PM  <  0 

U(PM)  =  1  for  PM  2  0 

Note  that  this  function  is  an  experimental  function  but  due  to  the 
division  by  V  PM  increases  rapidly  as  PM  approaches  zero.  Also 
due  to  the  folding  of  negative  control  error  values  into  positive 
errors  only  a  one  tailed  test  is  required  in  testing  for  significance. 
Time  available  did  not  permit  development  of  the  sample  statistics 
using  this  density  function.  Thus  to  facilitate  presentation  of  the 
results,  the  subsequent  analyses  will  involve  the  mean  of  the 
score  (PM)  distribution. 

Analysis  Step  2. 

Analysis  Step  2  is  the  second  step  illustrated  in  Table  8 
in  which  performance  of  all  flights  performed  by  each  subject  are 
analyzed.  The  performance  data  are  categorized  by  three  factors: 
sector,  error /error  rate  cell,  and  treatment,  and  are  written 
into  computer  files  for  subsequent  analysis. 

Results  of  Step  2  are  shown  in  Figures  4,  5,  6  and  7 
each  of  which  corresponds  to  a  particular  treatment.  For  instance. 
Figure  4  presents  the  results  for  the  command/night  treatment 
combination  for  Subject  1  while  Figure  5  gives  the  results  for 
the  conventional/night  treatment  also  for  Subject  1 .  The  figures 
are  believed  to  be  self  explanatory  indicating  sector,  error/error 
rate  cell  indicated  under  the  column  marked  "code",  the  number 
of  occurrences  for  that  subject  along  with  the  mean  PM  score 
and  the  score  squared.  The  number  of  occurrences  (N)  has  a 
maximum  value  of  3  since  each  subject  flew  exactly  three  flights 
in  each  treatment.  However,  N  can  be  less  than  three  if  the 
corresponding  condition  did  not  arise  during  that  particular  flight. 
While  some  apparent  differences  can  be  detected  by  examining 
different  performance  scores  for  different  treatments,  the  purpose 
of  this  analysis  step  was  not  to  compare  performance  effects  for 
individuals  but  rather  to  provide  an  intermediate  step  towards 
comparison  of  performance  effects  for  all  subjects.  The  data  here 
are  presented  as  documentation  of  tnat  step. 
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ifcCK* 

m. 

N 

t'Erri  :-j.: 

W! 

1 

1 

v> 

0.047l32@ 

8.t>Xt223) 

1 

4 

3 

8.1137871 

8.8129293 

2 

1 

3  • 

0.8163684 

0.0082677 

2 

2 

1 

0.0117095 

0.080137! 

2 

4 

2 

0.003.73% 

8.0000140 

3 

1 

3 

8.0289132 

8.0008368 

3 

4 

3 

8.0124943. 

8.00101561 

4 

1 

3 

8.0320681 

8.8010284 

4 

2 

1 

8.8365394 

8.0013351 

4 

3 

1 

8.8479375 

0.0622980 

4 
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Figure  4.  Sector  vs.  error/error  rate  (code)  analysis: 
Subj:  #1  ,command/night. 
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Figure  5.  Sector  vs.  error/error  rate  (code)  analysis: 
Subj:  #1ponventional/night 
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Figure  6.  Sector  vs.  error/error  rate  (code)  analysis 
Subj  #1 ,  command/day. 
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Figure  7.  Sector  vs  .  error/error  rate  (code)  analysis 
£ubj  #1 .  cpnventional/day 


Analysis  Step  3. 

This  part  of  the  analysis  was  to  combine  performance  data 
over  all  subjects  for  each  category  cell  and  to  calculate  the  mean 
score  and  score  variance. 

Results  of  the  analysis  are  arrayed  in  several  ways. 

First,  the  mean  score  and  score  variance  for  both  glide  path 
and  angle  of  attack  control  for  all  subjects  are  arrayed  against 
sector,  error/error  rate,  and  treatment  as  shown  in  Figures  8 
and  9.  These  figures  present  data  for  Sectors  1  and  2. 

Second,  as  shown  in  Figure  10,  performance  data  are 
organized  by  error  and  error  rate  versus  treatment  categories 
for  Sectors  1-8  inclusive  (Sector  9  was  eliminated  due  to  the 
large  penalty  error  associated  with  it.  Also  Figures  11  thru  18 
show  similar  data  for  Sectors  1  thru  8  respectively.  A  similar 
presentation  of  angle  of  attack  control  is  given  in  Figures  19,  for 
sectors  (1  -  8).  Note  that  the  error  and  error  rate  categories 
correspond  to  glide  path  error  and  error  rate  in  all  the  figures 
so  that  even  though  the  scores  for  angle  of  attack  control  for 
a  particular  treatment  are  presented,  the  error  and  error  rate 
categories  are  determined  for  the  glide  path  error  and  error 
rates.  This  permits  examination  of  the  effect  on  angle  of  attack 
control  of  ah  error  condition  in  glide  path  control. 

Returning  now  to  Figure  10,  it  is  seen  that  the  command 
display  supports  improved  performance  over  that  of  a  conventional 
display  but  the  improved  scores  (i.e.,  lower  penalty  scores)  are 
achieved  only  for  day  light  conditions  and  apparently  only  under 
error  and  error  rate  conditions  1,  3,  and  4.  Apparently  little  is 
gained  with  the  command  display  when  the  glide  path  error  is 
large  but  the  error  rate  is  small.  Also,  the  improved  performance 
of  the  command  display  over  conventional  display  is  not  as  evident 
in  night  landings  as  it  is  in  day  landings. 

But  averaging  scores  over  Sectors  1  thru  8  does  not  tell 
the  whole  story.  Turning  to  Figure  11  which  gives  scores  for 
the  terminal  sector,  it  is  again  seen  that  the  command  display 
provides  improved  performance  over  the  conventional  display  for 
day  landings  but  not  for  night  landings.  Note  that  error  and  error 
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Figure  8.  Sector  vs.  error/error  rate  vs.  treatment  analysis. 
All  subjects. 
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9.  Sector  vs.  error/error  rate  vs.  treatment  analysis 
All  subjects. 
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Figure  1 1.  Glide  path  control  scores: 

treatment  vs.  error/error 
rate.  (Sector  1) 
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Figure  IS  Glide  path  control  scores: 

treatment  vs .  error/error 
rate .  (Sector  2) 
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Figure  13.  Glide  path  cbntrol  scores: 

treatment  vs .  error/ error 
rate.  (Sector  3) 
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Figure  14.  Glide  path  control  scores: 

treatment  vs .  error/error 
rate.  (Sector  4) 
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Figure  15.  Glide  path  control  scores: 

treatment  vs .  error/ error 
rate.  (Sector  5) 
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Figure  16.  Glide  path  control  scores: 

treatment  vs .  error/error 
rate.  (Sector  6) 
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Figure  17.  Glide  path  control  scores: 

treatment  vs .  error/error 
rate.  (Sector  7) 
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Figure  18.  Glide  path  control  scores: 

treatment  vs.  error/error 
rate.  (Sector  8) 
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rate  (Category  2)  (large  error  -nd  small  error  rate)  did  not 
occur  in  Sector  1  under  the  day  condition.  When  large  glide 
path  errors  occurred,  large  error  rates  also  occurred  in  the 
terminal  sector.  The  effect  of  the  improvement  with  the 
command  display  for  day  landings  is  more  apparent  in  Sector  1 
than  it  is  when  the  data  is  averaged  over  all  sectors. 

Data  for  glide  path  control  arrayed  for  Sector  2  is 
shown  in  Figure  12.  Sector  2  is  the  sector  that  preceeds 
Sector  1 ,  the  terminal  sector,  and  it  is  useful  to  compare 
flight  performance  just  prior  to  terminal  control  with  the 
terminal  control  performance.  We  see  a  consistent  trend  in 
the  efFect  of  the  treatment  factors  but  the  errors  are  considerably 
smaller  than  they  are  for  Sector  1 .  Since  the  mean  score  values 
in  Sector  2  are  similar  to  those  obtained  for  Sector  1  thru  8, 
it  is  apparent  that  overall  performance  along  the  glide  path 
degrades  in  almost  all  categories  in  Sector  1 .  This  is  probably 
due  to  attempts  by  the  pilot  to  make  last  minute  corrections  of 
flight  path  errors.  This  characteristic  often  occurs  in  terminal 
control  problems  but  is  not  typical  for  experienced  pilots.  It  is 
typical  for  less  experienced  operators  who  anticipate  the  termination 
of  the  problem  and  attempt  to  reduce  errors  rapidly  by  increasing 
the  control  gain  -  frequently  resulting  in  an  unstable, divergent 
control  policy.  Further,  it  is  interesting  to  note  that  in  Sector  1 
where  small  error  and  small  error  rates  exist,  the  performance 
penalty  scores  are  small  for  treatments:  command/day  and 
conventional,  command/night  conditions.  This  suggests  that  as 
noted  previously,  when  Sector  1  is  entered  with  small  errors  and 
error  rates,  presumably  the  result  of  an  experienced  pilot's 
control  policy,  the  control  policy  is  not  changed  during  the  final 
sector. 


An  auto  correlation  analysis  was  performed  to  determine 
the  correlation  of  glide  path  control  scores  shifted  T  samples  from 
each  other.  T  was  varied  from  1  to  10.  Since  samples  were  taken 
at  a  rate  of  30  times  a  second,  each  shift  is  equivalent  to  a  time 
difference  of  t/30  seconds.  It  was  expected  that  correlations  would 
be  high  for  small  T  with  a  reduction  in  the  correlation  coefficient 
for  increasing  T.  Results  showed  that  correlation  for  T=1  was 
high,  being  in  the  order  of  .95.  But  correlation  values  for  larger  t 
shifts  (2  through  10)  dropped  to  a  low  value  -  in  the  order  of  .005. 
Such  a  rapid  reduction  in  correlation  coefficient  values,  with  in¬ 
creasing  T,  cannot  logically  be  attributed  to  the  speed  of  response 
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of  the  pilot  because  it  is  known  that  control  elements  were  held 
constant  for  longer  periods,  e.g.,  in  some  instances  the  throttle 
was  maintained  at  a  constant  level  for  several  seconds.  Thus 
there  is  strong  evidence  that  the  system  performance  measure, 
which  adjusts  the  reference  control  as  a  function  of  the  state 
variables,  presents  independent  problems  to  the  pilot  at  each  sample 
Additional  study  is  required  to  investigate  this  issue  since  we  can¬ 
not  expect  a  pilot  (or  other  human  operator)  to  respond  to  each 
sample;  but,  if  each  sample  is  an  independent  test,  then  each  flight, 
which  consists  of  many  samples,  will  contain  many  independent  tests 
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CONCLUSIONS  CONCERNING  CARRIER  LANDING  MEASURES 


The  method  of  measuring  aircraft  landing  performance 
by  examining  how  well  the  pilot  (operator)  controls  derivatives 
of  the  error  as  a  function  of  aircraft  state  (i  -  e . ,  the  error  and 
error  rate)  appears  to  be  feasible  computationally  and  also 
offers  several  attractive  features  for  the  researcher.  One 
feature  is  that  scores  can  be  obtained  from  performance  over 
short  intervals  of  time  (intervals  which  are  in  the  order  of  the 
human  response  which  is  in  the  range  of  .2  to  .5  seconds. 

This  frequent  evaluation  permits  scoring  in  system  state  cells  - 
such  as  the  sector  and  error  and  error  rate  cells  used  here, 
as  well  as  states  that  reflect  status  of  other  control  channels 
and  perhaps  status  of  secondary  tasks.  Also,  the  performance 
data  caui  be  arrayed  in  individual  treatment  combinations  or  can 
be  summarized  over  any  treatment  combinations  and  subjects  desired. 
Thus,  the  performance  measurement  method  provides  a  sensitive 
measure  since  the  effect  of  a  treatment  factor  which  influences 
performance  in  only  part  of  a  mission  or  influences  performance 
in  only  some  system  states  cam  be  readily  determined  since  per¬ 
formance  in  each  cell  can  be  evaluated  independently. 

It  should  also  be  noted  that  any  non-linear  control 
characteristics  exhibited  by  the  pilot  can  be  investigated  using 
this  performance  measurement  method.  The  method  permits 
evaluation  of  such  non-linear  characteristics  such  as  control 
saturation  (e.g.,  maintaining  aircraft  roll  angle  at  a  fixed  level 
until  the  desired  heading  is  achieved),  asymmetrical  control 
(e.g.,  a  tendency  to  reduce  all  errors  to  a  small  positive  (or 
negative)  error  before  final  correction),  and  offset  (e.g.,  when 
a  small  but  non  zero  error  is  maintained  until  the  terminal  portion 
of  the  flight  is  reached  where  the  small  error  is  zeroed).  It 
should  be  noted  that  the  mechanism  for  generating  reference  control 
has  been  developed  as  shown  in  Connelly  &  Loental,  1974  to 
include  the  non-linear  control  rules.  And  further,  the  "resources- 
to-go”  function  (0)  can  be  developed  tc  indicate  the  stability  of 
such  non-linear  systems . 

Score  comparisons  over  small/large  error  and  error 
rates  suggest  that  the  pilots  may  be  using  non-linear  control 
policy.  The  score  differences  suggest  that  a  more  fine  grained 
quantifying  of  scores  in  error/error  rate  cells  may  reveal  the 
non-linear  pilot  control  role. 
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RECOMMENDATIONS  CONCERNING 
CARRIER  LANDING  MEASURES 

There  are  several  additional  analyses  that  should  be 
completed j  but  limited  time  did  not  permit  completion  of  the 
analysis  here.  These  analyses  are: 

1 .  The  investigation  of  the  rate  at  which  independent 
control  problems  that  can  be  presented  to  an 
operator  (pilot).  We  can  argue  that  since  the 
pilot  has  the  opportunity  to  and  is  capable  of 
rapidly  selecting  the  control  that  will  give  him 
zero  penalty;  therefore,  the  data  samples, 
separated  by  a  sufficiently  large  T,  are  independent 
control  problems.  The  question  is  however:  How 
large  should  T  be  to  insure  independent  control 
problems . 

2.  There  is  a  need  to  investigate  models  of  different 
order  than  second  order  to  evaluate  the  properties 
of  the  resulting  measures. 

3.  There  is  a  need  to  test  the  control  law  to  determine 

if  non-rl inear  control  policies  are  used  and  to  determine 
if  the  measure  should  be  modified  to  detect  the  effect 
of  any  non-linear  control  policies. 

4.  There  is  a  need  to  identify  some  flight  data  as 
expert  data  so  that  the  measure  functions  can  be 
developed  from  these  flights.  That  methodology 
would  use  a  conceptually  different  approach  then 

the  one  used  here  and  would  involve  the  development 
of  the  resource-to-go  (fl)  by  backwards  integration 
of  an  assumed  performance  function  from  the 
terminal  point  back  to  each  state  in  the  problem 
space . 


55 


AIR-TO-GROUND  MEASUREMENT  PROBLEM 


The  air-to-ground  performance  measurement  problem 
is  to  develop  a  measurement  system  for  the  positioning  of  an 
aircraft  to  launch  a  weapon  such  as  a  rocket  or  bomb,  or  to 
fire  guns  at  a  ground  target.  Particular  aircraft  approach 
maneuvers  may  be  desirable  to  minimize  aircraft  detection  by 
enemy  forces  and  to  minimize  risk  to  the  attacking  aircraft. 

The  problem  is  characterized  by  the  existence  of  multiple 
mission  segments  where  the  terminal  conditions  on  one  segment 
are  the  initial  conditions  for  the  subsequent  segment. 

Furthermore,  the  aircraft  state  at  a  successful  weapons 
launch  is  not  a  point  in  state  space  but  rather  is  a  hypersurface 
of  the  state  variables.  This  hypersurface  reflects  the  functional 
relationships  among  state  variables  that  can  exist  at  weapons 
launch  that  permit  a  direct  hit  on  the  target;  The  function 
representing  the  surface  is 

G(X)  =  K 

where  X  is  a  vector  of  state  variables.  However,  one  or  more 
isolated  points  on  this  hypersurface  may  be  preferred  because  of 
several  possible  reasons:  one  reason  is  that  a  particular  launch 
point  may  result  in  least  effect  of  variations  in  weapons  systems 
character istics ..  A  launch  point  may  also  be  favored  because  the 
aircraft  flight  path  leading  to  that  point  may  minimize  risk  to  the 
aircraft. 


Yet  another  characteristic  of  this  problem  is  that  a 
successful  flight  path  (resulting  in  a  target  hit)  and  an  unsuccessful 
flight  path  (resulting  in  a  substantial  miss)  may  be  similar.  For 
example,  aircraft  motion  along  a  preferred  path  but  combined 
with  an  early  or  late  weapons  launch  can  lead  to  a  substantial 
miss.  Measurement  of  this  type  of  error  is  conceptually  different 
from  measurement  of  flight  path  errors. 

Therefore  in  the  air-to-ground  weapons  launch  problem  it 
is  not  possible  to  measure  performance  by  establishing  an  isolated 
reference  flight  path  and  scoring  flights  according  to  the  deviation 
from  that  reference  flight  path.  This  is  because: 
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1 .  There  are  many  possible  initial  (entry)  states  for 
each  segment  and  there  must  exist  a  preferred 
solution  from  each  entry  point.  (Actually  a 
preferred  solution  exists  from  every  point  in  the 
state  space  -  at  some  points  the  preferred  solution 
may  be  to  abort  the  attempt  and  try  again.) 

2.  Flight  corrections  toward  suitable  terminal 
(weapons  launch)  conditions  from  the  present 
state  must  be  scored  as  correct  when  it  is 
actually  possible  for  a  suitable  correction  to 
be  made. 

3.  Flight  paths  leading  to  a  successful  launch 
(i.e.,  a  target  hit)  but  not  passing  through  the 
favored  release  point  must  be  scored  according 
to  that  result  and  also  scored  according  to  the 
flight  path  (e.g.,  a  hit  may  have  been  achieved 
but  the  flight  path  used  was  not  ideal  from  an 
aircraft  risk  or  low  variance  weapon  performance 
viewpoint). 

As  a  result,  it  is  recommended  that  a  system  performance 
measure  be  developed  for  each  mission  to  be  used.  The  theory 
of  system  performance  measures  was  presented  previously  and 
applied  to  the  carrier  landing  problem .  However,  the  procedures 
for  implementing  the  measure  for  the  air-to-surface  weapons 
launch  would  be  quite  different.  This  is  because  there  was  no 
flight  designated  as  superior  that  could  be  used  to  develop  a  reference. 
Instead  a  reference  function  was  selected*  For  the  air-to-surface 
measurement  problem,  it  is  recommended  that  a  systematic  pro¬ 
cedure  be  used  to  establish  the  reference  summary  performance 
measure  and  then  to  synthesize  the  system  performance  measure. 

The  recommended  procedure  is  as  follows: 

1 .  Develop  mission  segments  and  segment  entry  criteria 
such  as  those  specified  by  Vreuls  &  Sullivan  (1982). 
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2.  Determine  for  all  segments  and  especially  the  segment 
ending  in  weapon  release  or  weapon  firing,  the  hypersurface  leading 
to  success  and  the  resulting  miss  from  deviations  from  that  surface. 
This  can  be  done  by  analysis  of  weapons  data  and  results  compared 
to  results  of  simulator  flights. 

3.  Develop  a  summary  measure.  One  part  of  a  summary 
measure  will  likely  be  the  weapon  miss  distance  (i.e.,  the  function 
developed  in  Step  2  above).  However,  there  are  other  factors  to 
consider  in  the  evaluation.  Smooth  coordinated  flights,  not  requiring 
rapid  corrections  at  the  end  of  a  sector,  and  low  risk  flight  as  well 
as  accurate  and  timely  navigation  are  also  important.  Thus  to 
develop  a  summary  performance  measure  it  is  recommended  that 
test  flights  be  flown  demonstrating  a  variety  of  performance  levels. 
These  flight  demonstrations  should  be  recorded  so  they  can  be 
presented  to  a  group  of  subject  matter  experts  for  a  relative 
ranking  according  to  preference.  Presumably  time  histories  of 
selected  state  variables  plotted  in  "strip  line"  type  plots  would 
serve  as  a  suitable  demonstration  device. 

It  is  expected  that  some  disagreement  would  occur 
among  the  experts  as  to  the  preferred  relative  ranking  of  the 
demonstrations.  Such  disagreements  usually  indicate  a  lack  of 
an  important  variable  in  the  documentation  (e.g.,  the  state  space 
needs  to  be  increased  y,  or  the  mission,  specification  has  not 
been  adequately  defined  (e.g.,  the  mission  tactics  are  to  be  changed 
because  of  the  available  jamming  of  defensive  equipment,  or  the  state 
of  aircraft  ammo  and  fuel  supply).  These  disagreements  tend  to 
enhance  the  understanding  of  the  problem  and  typically  result  in 
mission  parameters  being  introduced  into  the  summary  measure. 

A  comprehensive  summary  measure  is  then  developed 
as  a  function  of  the  state  variables  that  when  applied  to  the 
demonstration  missions  results  in  the  same  relative  ordering  as 
the  experts  provided.  (Note  that  this  approach  is  quite  different 
from  asking  subject  matter  experts  what  they  think  is  important  to 
performance  measurement  and  then  relying  on  that  judgment. 

Instead,  the  subject  matter  experts  are  asked  to  order  demonstrations 
according  to  performance  preference.)  Mission  parameters  permit 
revised  ordering  based  on  mission  conditions  such  as  "if  condition 
X  is  true,  then  the  summary  measure  function  is  F-j(S),  if  condition 
X  if  false  then  the  function  is  F2(X)." 
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PMA  uses  its  MAP  Processor  (Connelly  &  Johnson, 
1981)  to  develop  the  summary  performance  measure  function. 

4.  Based  on  the  summary  measure,  the  system  per¬ 
formance  measure  is  synthesized  by  backwards  integration  of 
the  summary  measure  along  solution  trajectories  in  accordance 
with  the  measurement  equations  given  previously.  The  backwards 
integration  gives  the  "resources  required  to  complete  the  mission" 
(0)  described  previously. 

The  system  performance  measure  is  formed  as  the 
sum  of  the  rate  of  change  of  0  ,  i.e.,  d0/dt  plus  the  rate  of 
change  of  the  resources  used  according  to  Equation  (  6  ).  This 
is  equivalent  to  specifying  for  each  point  in  the  state  space  a 
reference  set  of  differential  equations  which  provide  the  reference 
change  in  the  aircraft  state  vector  as  well  as  the  penalty  for 
deviation  from  that  reference  -  according  to  the  summary  per¬ 
formance  measure. 

The  system  performance  measure  permits  continuous 
(or  sampled)  evaluation  of  flight  performance  at  each  point  in  the 
state  space. 
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COMPARISON  OF  PILOT  PERFORMANCE  EVALUATION 
WITH  LINEAR  PILOT  MODELS  AND 
SYSTEMS  PERFORMANCE  MEASURE 


A  comparison  of  the  nature  and  applications  of  linear 
pilot  models  and  systems  performance  measures  is  developed  in 
this  section.  First, the  nature  of  the  pilot  models  is  considered 
along  with  its  application  parameters.  Next  the  models  are  con¬ 
trasted  with  system  performance  measures  in  regard  to  applica¬ 
tions.  And  finally,  the  relationship  between  the  two  entities  is 
developed . 


An  application  of  linear  pilot  modeling,  given  by 
Jewell  1981,  provides  identification  of  pilot  dynamics  in  a  carrier 
landing  problem.  A  carrier  landing  is  a  multi-loop  control  task 
requiring  control  of  glide  path  error  via  the  throttle  and  control 
angle  of  attack  error  via  the  elevator.  Jewell  (1981)  identified  the  pilot 
transfer  function  for  the  glide  path  and  angle  of  attack  channels. 

He  did  not  consider  the  lateral  aircraft  control  channel.  Results 
reported  by  Jewell  were  the  identification  of  the  transfer  function 
gain  coefficients  of  the  pilot  glide  path  and  angle  of  attack  control 
along  with  that  of  the  crossover,  or  coupling,  from  throttle 
movement  to  a  compensating  elevation  movement. 

In  order  to  understand  these  results  and  relate  them 
to  performance  evaluation  of  the  carrier  landing  task,  it  is 
first  necessary  to:  characterize  the  pilot  modeling  methodology, 
list  its  assumptions,  and  relate  its  capabilities  to  that  required 
for  the  required  performance  assessment. 

The  linear  pilot  modeling  method,  which  seeks  to 
characterize  pilot  transfer  characteristics  as  a  function  of  fre¬ 
quency.  is  a  construct  based  on  linear  control  theory.  The 
modeling  theory  has  been  well  developed  and  is  documented  in 
numerous  references  including  the  classic  reference:  McRuer  and 
Krendel  1957.  The  assumptions  required  to  develop  and  use  the 
model  are: 


1 .  Pilot  (operator)  outputs  are  only  a  function  of 
the  inputs. 

2.  Inputs  are  errors  which  (ideally)  are  to  be 
reduced  to  zero. 

3.  */''viel  output/input  function  is  a  linear  function, 

i.e.,  is  not  a  function  of  the  error  or  error 
rate  magnitude  or  other  flight  variables. 

4.  Pilot  output/input  function  is  constant  over 
intervals  that  are  much  longer  than  the  air¬ 
craft  time  constants. 

The  linear  pilot  model,  hearafter  referred  to  as  the  ’pilot 
model’,  is  seen  to  be  a  process  model  with  parameters  derived 
from  test  data.  If  the  model  is  to  be  used  to  evaluate  pilot 
or  system  performance,  the  parameter  values,  once  determined, 
must  be  related  to  system  performance  in  a  separate  analysis. 
Thus,  the  model  is  not  an  end  in  itself,  but  could  be  an  tntei — 
mediate  step  in  development  of  a  performance  measure. 

But  the  modeling  technique  has  several  fundamental 
limitations  when  used  to  support  a  performance  measurement 
system.  These  limitations  are: 

1 .  The  model  cannot  detect  or  represent  rapid 

changes  in  pilot  control  strategy. 

2.  The  model  cannot  represent  non-linear  control 
characteristics  -  for  instance,  control  policy 
differences  as  a  function  of  error  and  error 
rate,  which  typically  occur  for  large,  inter¬ 
mediate  and  small  errors,  cannot  be  represented. 

3.  The  model  cannot  represent  outputs  that  are  not 
functions  of  input  flight  errors.  For  instance, 
present  outputs  might  be  produced  by  the  pilot 
as  a  function  of:  anticipated  future  events 
(e.g,,  expected  wind  gust  near  surface  or 
expected  change  in  aircraft  configuration),  pre¬ 
planned  open  loop  control  strategies,  and  other 
flight  factors  that  are  not  flight  errors. 
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4.  Flight  paths  are  interpreted  only  according  to  linear 
control  theory.  Thus,  for  instance,  using  linear 
control  theory  and  referring  to  Figure  20,  flight 
path  A  would-  be  interpreted  as  resulting  as  a 

high  gain,  "tight",  pilot-fatiguing  control  policy. 

And  flight  path  B  would  be  interpreted  as  a  low 
gain,  "loose",  pilot-relaxed  control  policy.  But 
flight  path  B  might  also  result  form  the  pilot's 
selection  of  a  different  reference  flight  path  (i.e., 
path  B  is  the  reference)  which  is  flown  without 
large  errors.  Thus,  the  assumption  of  a  linear 
model  and  the  assumption  that  the  pilot  is  tracking 
only  the  reference  flight  path,  i.e.,  glide  path, 
forces  interpretation  of  Flight  trajectories  to  be 
one  of  a  restricted  set  of  linear  flight  control 
policies  —  none  of  which  may  correspond  to 
the  pilot’s  actual  flight  control  policy. 

5.  The  model  assumes  that  the  pilot  is  engaged  in 

a  tracking  task  that  has  no  beginning  and  no  ending. 
However,  most  real -world  tasks  have  a  definite 
beginning  where  initial  control  is  important  and 
a  specific  end  condition  where  specified  conditions 
are  important  to  performance  evaluation. 

6.  The  model  requires  disturbances  to  the  controlled 
system  in  order  to  force  pilot  control  actions.  In 
flight  simulation  studies,  where  disturbances  can 
be  specified  as  desired,  disturbances  are  often 
tailored  to  satisfy  model  development  requirements 
rather  than  representing  likely  disturbances  in 

the  real  world, 

7.  The  use  of  the  linear  pilot  model  for  an  analysis 
tool  has  led  to  concentration  on  the  development  of 
displays  providing  flight  error  signals  -  i.e., 
displays  that  convert  the  flight  control  task  into 

a  tracking  task.  But  in  many  flight  tasks  such 
as  air-to-surface  weapons  release  tasks,  the 
flight  control  problem  is  one  of  directing  the 
aircraft  from  its  present  state  to  a  weapon  re¬ 
lease  state  (actually  a  hyper-surface). 
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Other  types  of  displays  such  as  situation  displays 
which  present  the  "total"  situation  to  the  pilot  or 
displays  that  present  the  preferred  incremental 
solution  from  the  present  state  should  be  evaluated. 

System  Performance  Measures 

Comparison  of  the  linear  pilot  models  '*uth  system  per¬ 
formance  measures  is  facilitated  by  reference  to  the  preceding 
section.  It  is  seen  that  the  linear  pilot  model  is  not  a  measure 
but  a  construct  that  might  support  a  performance  measure.  The 
model  has  the  advantage  of  being  a  well  developed  technology  so 
that  considerable  data  and  application  experience  is  available  for  it. 
However,  its  application  areas  are  limited  by  the  required  assumptions 
and  other  limiting  problems  cited  previously. 

In  contrast ,  a  system  performance  measure  is  not  a 
pilot  model  but  is  a  continuous  (or  discrete)  measure  permitting 
evaluation  of  instantaneous  (or  sampled)  performance  demonstrations. 

An  SPM  measures  performance  based  on  only  the  control  outputs 
and  system  state  variables.  As  a  result  no  control  input  structure 
is  assumed.  No  linearity  of  control  is  assumed.  The  SPM  is  not 
concerned  with  the  inputs  used  -  it  is  concerned  with  the  control 
outputs  and  the  performance  implication  of  those  outputs.  Further, 
the  assumption  that  the  inputs  are  tracking  errors  is  not  required. 

An  SPM  can  measure  performance  of  a  rapidly  changing  control 
policy,  a  non-linear  control  policy,  an  open  loop  pre-planned  control 
policy,  an  non-linear  control  policy,  or  a  control  developed  by  any 
rule  the  pilot  chooses.  Further,  the  measure  does  not  require  use 
of  flight  disturbances  -  such  as  wind  simulation.  SPMs  are  developed 
for  terminal  control  tasks;  they  are  not  readily  applied  to  tracking 
control  tasks. 


Thus,  it  is  seen  that  the  linear  pilot  model  and  an  SPM 
are  different  entities  and  have  different  applications.  Yet  there  is  sin 
interesting  relationship  between  them.  The  relationship  is  established 
when  the  SPM  is  used  to  develop  a  non-linear  pilot  model.  A  non¬ 
linear  model  is  required  when  large  and/or  small  error  magnitudes 
are  to  be  controlled  -  these  are  the  cases  where  the  linear  pilot  model 
fails.  The  method  is  to  design  a  non-linear  pilot  model.  (NLPM)  and 
then  use  the  Linear  Pilot  Model  (LPM)  to  fix  the  characteristics  of 
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the  NLPM  when  errors  are  of  intermediate  magnitude  where  linear 
pilot  control  policies  are  expected.  Once  that  portion  of  the  model 
is  completed,  it  is  extended  to  represent  the  non-linear  control 
associated  with  large  and  small  errors. 

To  show  this  relationship,  consider  now  construction 
of  a  NLPM.  First,  a  convergence  indicator  function  (  $  )  is  selected 
as  described  in  the  previous  sections.  If  a  pilot  control  is  observed, 
then  the  rate  of  change  of  9  can  be  calculated  as: 

9  (X  (t)  )  =  ~6  }  (40) 


and  can  be  recorded.  If  there  are  no  disturbances,  one  would  expect  that: 


6  (X  (t)  )  <  0 


(41) 


almost  everywhere.  With  disturbances  9  will  have  both  positive  and 
negative  values.  Now  note  that  since  9(X)  is  positive-definite,  the 
ratio 


Z(X  (t)  ) 


-0  (X  (t)  ) 

9  (X  (t)  ) 


(42) 


will  be  positive  whenever  9  is  negative. 

Now  suppose  numerous  tests  of  a  control  task  are  con¬ 
ducted  and  the  resulting  flight  data  recorded.  A  model  of  the  control 
strategy  demonstrated  in  that  task  can  be  developed  by  determining 
the  function  Z(X)  from  the  above  equation.  Two  methods  are  available 
for  developing  the  Z (X)  function.  One  method,  which  is  described 
here  first,  is  to  conduct  the  flight  tests  without  artificial  disturbances 
and  fit  Z(X)  to  the  observed  data.  The  other  method,  described 
subsequently,  is  to  use  a  model  construct  such  as  the  linear  pilot 
model,  which  is  derived  using  disturbances,  to  determine  Z(X)  in  the 
regions  of  (X)  where  the  model  is  valid  and  then  extend  Z(X)  into 
regions  where  the  linear  model  is  not  valid  by  a  data  fitting  procedure. 

Considering  the  first  method,  assume  that  data  for  multiple 
flight  demonstrations  have  been  recorded.  The  function  0(X)  is  known 
since  it  was  selected  and  the  value  of  g  (X  (t)  )  can  be  calculated 
along  flight  trajectories  from  the  data.  Next, the  function  Z(X)  is 
determined  using  a  least  squares  fit  to  the  data.  Finally,  the  polarity 
of  Z(X)  is  observed.  If  Z(X)  is  positive  everywhere  (except  perhaps 
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at  the  origin),  it  is  excepted  for  the  NLPM  function.  If  Z(X)  is 
negative  in  large  regions  of  the  state  space,  it  may  be  necessary 
to  select  a  different  9  function.  Finally^  if  Z(X)  is  negative  only 
occasionally  or  in  small  regions  of  the  state  space,  Z(X)  is 
approximated  by  a  function  g(X)  such  that: 


9(X) 

g(X) 


Z(X) 


*  (X) 
9  (X) 


when  Z(X)  >  0 


Q(X)  when  Z(X)  <  0 


where  Q(X)  is  a  positive  definite  function.  Then,g(X)  is  used  as 
the  NLPM  function. 

The  purpose  of  developing  the  approximation  g(X)  to 
Z (X)  is  to  guarantee  a  stable  control  model  in  the  sense  that  9 
indicates  a  stable  control  system.  However, the  stability  of  some 
systems  is  state  related.  For  instance,  a  system  might  converge 
until  a  small  error  is  achieved  (typically  where  the  pilot  has 
difficulty  in  extracting  error  rate  information  from  the  display) 
where  a  limit  cycle  is  established.  In  these  instances  g(X)  can 
be  modified  to  provide  convergence  to  the  limit  cycle  and  establish 
the  limit  cycle  itself.  Stability  here  is  not  limited  to  stability  of 
a  linear  control  system.  A  model  for  any  stable  control  system 
can  be  developed  by  choosing  or  deriving  an  appropriate  0(X) 
function. 


Using  g(X)  as  the  NLPM  function  results  in 


0(X)  =  -0(X)  g (X) 

where  9  is  negative  definite  because  both  9  and  g  are  positive  definite. 
It  now  remains  to  show  how  the  NLPM  control  rule  can  be  developed. 
Suppose,  for  illustration,  that  the  equation  of  motion  of  the  controlled 
element  can  be  written  in  the  form: 


XN  =  F(X)  +  U 


(43) 
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where  X.  are  system  state  variables  and  U  is  a  control  element. 
(Only  oh4  control  element  is  used  here  for  illustration,  but  the 
method  applies  to  multi  input  control  problems.) 

By  definition 


N 


i 


X 

i 


(44) 


11 


X2  + 


(F(X)  +  U) 


(45) 


Solving  for  U  provides  the  NLPM  control  rule: 


U 


N-1 


*(X)  g(X)  -  F (X) 

^XN 


11 


(46) 


Now  corraider  the  second  way  of  constructing  the  NLPM 
using  results  gtven  by  McRuer  and'  Krendel,  1957.  According  to 
McRuer  and  Krendel,  when  the  controlled  element  transfer  function 
is  given  by: 


(47) 


where: 


Kq  is  a  given  constant  and  S  is  the  Laplace  operator, 
the  approximate  human  operator  traaafer  function,  i.e.,  the  linear 
pilot  model  is: 
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(48) 


S 


-TS 

e 


where 


Kp  is  a  constant  and  T  a  time  delay  factor. 


In  terms  of  state  notation  the  controlled  element 


equations  are 


X,  =  x2 


X2  *  KC  U 


and  the  control  equation  is 


U  =  “KP  X2  (t_T)  ‘ 


(40) 


Next,  since  X2  is  not  a  function  X, 
Equation  43  that 


we  conclude  from 


F(X)  =  0. 


(50) 


Substitution  of  U  from  Equation  (49)  into  Equation  (45)  and  solving 
for  g(X)  yields 


g(X) 


X0  (t-T) 


M. 

3X_ 


AL 


dx 


x. 


-0(X) 


Ttiis  formulation  produces  a  model  that  is  equivalent  to  that  of  the 
LPM  including  a  pilot  time  delay  function. 
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Extension  of  the  model  to  represent  the  effect  of 
limit  cycles  is  accomplished  by  defining  the  limit  cycle  path 
so  that: 


h(X)  =  1 

along  the  limit  cycle  path,  h(X)  <  1  "inside"  the  limit  cycle,  and 
h(X)  >  1 

"outside"  the  limit  cycle  as  shown  in  Figure  20.  Then  a  modified 
g  function  can  be  constructed  as: 


g^X)  =  ( — where  h(X)  >  0 

Thus ,  when 

h(X)  »  1 

g1^  a  9(X) 

But  when 

h(X)  =  1 

g1  (X)  =  0 

As  a  result 

9  =  -g1  (X)  0(X)  =  0  when  h(X)  =  1. 

Therefore,the  NLPIV1  provides  a  control  policy  that  will  attempt  to 
converge  the  lar^  errors  to  the  limit  cycle  but  will  also  diverge 
small  errors  to  the  limit  .cycle  as  shown  in  Figure  21 .  A  more 
complete  description- of  this  general  modeling  technique,  which 
includes  conditionsi  for  model  realization  and  an  application  to 
aircraft  control  problems.,,  is  given  in  Connelly  and  Loental,  1974. 


69 


The  purpose  of  this  presentation  is  to  show  a  relationship 
between  the  linear  pilot  modeling  technique  and  the  system  performance 
measure.  The  two  entities  are  different  and  are  used  for  different 
purposes.  A  relationship  does  exist  when  the  SPM  analytical  techniques 
are  used  to  develop  a  NLPM,  and  then  the  linear  pilot  model  along  with 
its  data  base  can  be  used  as  a  starting  point  for  the  NLPM. 
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APPENDIX  A 


PERFORMANCE  MEASURE 


PROOFS  AND  CONSTRUCTIONS 


Constructing  9  (x) 


The  resource-to-objective  function  9  (;<)  is  given  by: 

A 

0  (x(t)  )  =  j  -  F  (x(r),  U  (x(r)  )  dr  • 
t 

From  this  equation  one  must  have: 

d£  |  +  F  (x(t),  U  (x(t)  )  =  o 

dt  I  x(t) 

and  one  derives 

§£  (x(t)  )  ^  +  F  (x(t);  U  (x(t)  )  )  =  0  (A-1) 

along  a  solution  and  given  a  control  model  (g(x)  ): 

(x(t)  f  (x(t),  g(x(t)  )  )  +  F(x(t),  U(x(t)  )  *  0  (A-2) 

Now  let  9  (x)  be  the  solution  of  the  first  order  partial 
differential  equation: 

f£  (*)  f(x,  g(x)  )  +  F(x,  U(x)  )  *  0  (A-3) 

with  the  condition  9  (0)  =  0,  where  x  =  0  is  the  desired  terminal 
state.  Then  9  is  only  state-dependent  and  is  free  of  any 
particular  solutions.  Therefore,  along  any  solution  x(t),  one 
can  write  (A-2)  and  work  back  to  the  condition  (16)  in  the  body  of 
the  report. 

A  Necessary  and  Sufficient  Condition  for  a  Policy  to  be  Optimal 

A  necessary  and  sufficient  condition  for  a  policy  U*  to 
be  optimal  is  that 

0  (x,  U*)  =  min  0  (x,  U)  =  0 
U 


(A-4) 


The  purpose  of  this  section  is  to  make  this  result  more  precise 
by  indicating  a  proof.  In  order  to  establish  the  necessity,  a 
proof  by  contradiction  can  be  used.  Let  U*  =  g(x)  be  an 
optimal  policy  and  suppose  that  there  is  another  policy  U  =  g  (x) 
such  that  °  ° 


0  (x,  gn  (x)  )  <  0(x,  g*  (x)  )  =  o 


(A- 5) 


Then  d£  * 
dt 


+  F(x,  gQ(x)  )  <  0 


(A-6) 


and  thus 


3T  <xo  (t)  5  <  -FCxo®>  9o<xc®  5  > 


(A-7) 


where  x  (t)  is  a  trajectory  of  the  vehicle  (1)  in  the  main  body  of 
this  report  under  the  control  of  policy  Uq  =  g^  (x).  Integrating 


F  (XQ  (r),  gQ  (XQ  (r)  )  )  dr 


f  d 9* 

J  dr  ^Xo 


(r)  )  dr 
(A-8) 


Therefore 

I  CO  <  -C«*(x  (tj)  -  9*  (x  (o)  )  ]  =  0  *(x  (o)  )  =  I(U*) 

(A-9) 

since  0*  (x0  (tf)  )  =  0  at  the  terminal  state.  This  contradicts 

the  fact  that  U*  is  optimal. 

To  show  the  sufficiency,  assume  that  the  solution  time 
using  the  control  policy  U*  is  t^ .  The  integral  of  equation  (6) 
in  the  report  is: 


^1  *  N 

J  $  (X,  g*(x)  )  dr  =  J  ^  dr  +  j  F(x,g*(x)  )  dr  = 


(A— 10) 


A-2 


(A-11) 


where 


9*  (t^  =  0 


(A- 12) 


Also 

/  F(x,  g*(x)  )  dr  =  I  (U*;  tp  -  I(U*;  tQ) 
t 

o 


(A-13) 


where  I  (U*j  tQ)  =  0  (A- 14) 

Thus  the  cost  of  using  the  control  policy  U*>  is  represented  by 
I  (U*)  =  I(U*;  tp  and 

I  (U*)  =  8*  (tQ)  (A- 15) 


One  can  integrate  equation  (6)  in  a  similar  manner  assuming 
that  the  solution  time  for  another  control  policy  U  =  g(x)  is 
t-2 .  Thus 

f  0  <x-  )<*'■/  If  dr  +  f  F(x,  900  )  dr 

*t  t  ^  t 

o  o  o 

=  *(t2)  -  9*  (to)  +  I  (Uj  t2)  -  I  (U;tQ)  (A— 16) 
but  I(U;  tQ)  =  0  (A- 17) 


and  9*  (t_)  =  0  since  the  desired  state  is  the  origin  where  8  *  is 
zero.  Thus,  the  cost  of  using  the  control  policy  U  represented 
by  I  (U)  is 


A-3 


I  (U)  =  I  CU;  t  )  =  8*  -ft  )  + 

2  O 


0  (x,  g(x)  )  dr 


o  (A- 18) 

From  relation  shown  in  equation  (A-4),  one  has  0  (x,  g(x)  )  ^ 

0  and  clearly  the  integral  of  this  function  must  be  greater  than 
or  equal  to  zero.  Thus  one  concludes  that 

I  (U*)  ^  I  (U) 

which  is  what  was  set  out  to  be  proven. 

Developing  the  Performance  Measure 

for  a  Linear  Plaint  with  Quadratic  Control 

From  the  definition  of  U  *  U*  +  U,  where  U*  is  given 
by  equation  (A-4)  and  0  given  by  equation  (17).. 


0  Ax  -  ^  BR 
3  x  ax 


■’  sT  (KJ 


Qx  +  2 1* BR_1  bt  {U)T:Ui^-b-T^T 

(H)  T  +  2  4ljT  R4U  (f 


(A- 19) 


Then  1  T  ....  ,1TT  IT  IT 

0  =  -x  KA  x  +  -  x  A  Kx  -  -  x  Kx+^x  Qx 


^iUT  RAU  =  ~  A  UT  RAU 


(A-20) 


0  =  i  (U  -  U*)T  R  (U  -  U*) 


(A-21) 


where  .  ,*  -1  T  _ 

U*  =  -R  B  Kx 


(A-22) 


and  K  satisfies  the  Ricatti  equation 

KA  +  ATK  -  K  BR”1  BT  K  +  Q  =  0. 


A-4 


